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I.  INTRODUCTION 

Our  description  of  semi-Markov  processes  ^SMP,s)  is  heuristic 
and  unconventional.  We  postpone  a  formal  definition  of  a  distinguished 
state  and  a  discussion  of  regularity  conditions  to  following  sections 
Many  recondite  matters,  especially  those  that  arise  in  the  infinite 
state  case,  are  omitted. 

A  key  concept,  already  familiar  to  readers  acquainted  with  queueing 
theory,  is  that  of  an  imbedded  Markov  chain  connecting  the  regeneration 
points  of  a  stochastic  process.  Starting  from  a  regeneration  point, 
the  future  is  stochastically  independent  of  the  past.  All  distinguished 
states  correspond  to  regeneration  points;  thus,  they  have  the  Markov 
property.  In  the  imbedded  Markov  chain,  the  original  time  scale  of  the 
transitions  between  the  distinguished  states  is  replaced  by  a  discrete 
time  version  where  all  transitions  take  unit  time.  By  studying  the 
corresponding  semi-Markov  process,  defined  precisely  in  Sec.  3,  we 
recover  the  probabilistic  behavior  in  the  original  time  scale. 

I+  is  defined  to  be  the  set  of  distinguished  states,  assumed 
countable.  The  state  transitions  form  a  Markov  chain  with  transition 
probabilities  (p^),  where  direct  transitions  from  a  state  to  itself 
(e.g.,  p^  >  0)  are  allowed.  Given  that  an  i  *•  J  transition  is  about 
to  occur,  the  duration  of  the  transition  has  distribution  F^,  .  We 
define 


V'> 


pijFij(t) 


"i(t>  -IX/0- 

J 


4r  T 

Unless  otherwise  stated,  all  summations  will  be  over  I  and  all 
functions  vanish  for  negative  arguments. 


-3 


Thus  H.  is  th«  unconditional  distribution  of  the  time  elapsed  starting 
from  state  i  until  the  next  state  is  entered  (possibly  i  itself).  Note 
that 

(i)  a  one-state  SMP  is  a  renewal  process; 

(ii)  an  SMP  with  F^  degenerate  at  one  for  all  i,  j  is 
a  Markov  chain; 

(iii)  an  SMP  with  all  Fji  exponential  is  a  continuous 
time  countable-state  Markov  process. 

Many  problems  in  management  science  and  operations  research  can 
be  modeled  as  SMP's:  for  example,  queueing,  inventory,  and  mainte¬ 
nance  problems.  For  details,  see,  e.g.,  Pyke  [27],  Barlow  and  Proschan 
[1],  Fabens  [11],  and  ^inlar  [4].  For  proofs,  citations  of  earlier 
papers,  and  additional  topics  in  SMP's,  the  reader  should  consult  the 
reference  list.  Another  expository  paper  is  Janssen  [19]. 
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II.  DISTINGUISHED  STATES 


In  this  section  we  examine  precisely  what  is  meant  by  a  dis¬ 
tinguished  state.  Since  discussing  this  topic  in  an  offhand  manner 
could  result  in  confusion,  the  subject  is  treated  in  some  detail. 

For  each  sample  path  of  a  stochastic  process  X,  there  is  a  corre 
spondence  between  [t:  t  2  0]  and  a  set  of  states  S.  If  every  state 
in  S  is  required  to  have  the  Markov  property,  then  in  general  S  will 
be  uncountable  since  a  history  of  the  process,  or  at  least  the  rele¬ 
vant  portion  of  it,  must  be  part  of  the  state  definition.  However, 
all  that  we  require  of  S  is  that  it  have  an  appropriate  countable 
subset  I+  of  (distinguished)  states  having  the  Markov  property.  Thus 
for  a  state  to  be  a  candidate  for  I+,  it  must  correspond  to  a  regener 
ation  point,  but  we  do  not  require  that  all  states  corresponding  to 
regeneration  points  belong  to  I+. 

To  fix  these  ideas  concretely,  we  illustrate  them  with  examples 
from  the  M/C/l  queue  (Poisson  arrivals,  general  service  time  distri¬ 
bution,  single  channel).  For  many  purposes,  a  convenient  set  of  dis¬ 
tinguished  states  is  (0,  1,  2,  ...,  i,  ...}  where  state  i  signifies  i 
customers  in  the  system  and  a  service  has  just  been  completed.  Nate 
that  not  all  regeneration  points  are  included  in  this  set,  since  any 
time  the  system  is  idle  (empty),  it  is  at  a  regeneration  point.  Our 
choice  of  distinguished  states  conforms  to  our  requirements  because 
the  time  to  the  next  arrival  is  stochastically  independent  of  the 
time  elapsed  since  the  last  arrival.  In  general,  arrival  epochs, 
except  those  corresponding  to  the  start  of  a  busy  period,  are  not 
regeneration  points.  Thus  the  state  "i  customers  in  the  system  and 
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a  customer  has  just  arrived"  cannot  be  a  distinguished  state,  unless 
the  service  times,  are  exponential.  Through  the  use  of  so-called 
supplementary  variables,  we  can  define  every  state  in  the  original 
process  so  that  it  is  Markovian.  Each  state  is  then  a  couple  of  the 
form  (i,  u) ,  which  denotes  i  customers  in  the  system  and  the  customer 
being  processed  has  been  in  service  for  time  u.  Sometimes  supple¬ 
mentary  variable  techniques  are  useful  as  an  alternative  or  adjunct 
to  SMP  techniques;  see;  e.g.,  Cox  and  Miller  [5}. 

Returning  to  the  general  discussion,  we  require  the  distinguished 
states  to  be  defined  such  that  non-zero  holding  times  in  a  distinguished 

state  are  forbidden  but  instantaneous  transitions  among  the  distinguished 

!" 

states  are  allowed.  This  is  a  departure  from  the  setup  of  Pyke  [26"], 
although  the  two  formulations  are  essentially  equivalent.  Our  definition 
of  distinguished  state  is  natural  in  a  dynamic  programming  framework 
(see,  e.g.,  Denardo  and  Mitten  [9])  and  permits  a  graphic  representation 
of  SMP's  in  terms  of  networks  with  branch  nodes  (distinguished  states) 
and  stochastic  arc  lengths;  see  [133-  For  example,  traversing  an  arc 
could  correspond  to  a  customer  completing  service. 

★ 

To  remove  ambiguity  in  case  of  instantaneous  transitions,  we 

define  X+(t)  *  X(t+);  thus,  X+  is  right  continuous  and  last  distinguished 
^  1 

state  of  X  entered  in  [0,  t] ,  say,  is  well  defined.  Since  we  have 
prohibited  non-r.ero  holding  times  in  distinguished  states,  we  cannot 
allow  a  distinguished  state  to  correspond  to  a  nondegenerate  interval 
of  regeneration  points  (e.g.,  an  idle  period  in  an  M/G/l  queue).  Thus, 

*Sometimes  it  is  convenient  to  permit  instantaneous  transitions; 
see,  e.g.,  Denardo  [7]. 


to  exclude  an  Infinite  sequence  of  instantaneous  transitions  from  a 
state  to  Itself,  we  require  that  the  distinguished  states  be  defined 
such  that,  for  all  nondegenerate  intervals  (a,  b),  i  $  I+  =» 

Pfx(t)  i.  Vt  f  (a,  b)}  •  0.  For  example,  in  the  M/G/l  queue  it 
does  not  suffice  to  define  the  distinguished  state  0  as  0  customers 

in  the  system.  The  condition  that  a  service  has  just  been  completed 
must  be  added. 

Throughout  the  sequel,  "state"  refers  only  to  "distinguished 
state,"  necessitating  definitions  slightly  different  from  conventional 


usage . 
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III.  REGULAR  SMP'S 

Letting  N^(t)  denote  t!ie  number  of  times  state  j  is  entered  in 
the  half-open  interval  (0,  t] ,  we  obtain  the  Markov  Renewal  Process 

(MRP)  N(t)  -  (No(t),  Nj  (t) ,  ....  Nn(t)),  where  I+  -  [0,  1,  2 . n], 

possibly  with  n  ■  ®.  Let  Z(t)  be  the  last  distinguished  state  of  X+ 
entered  in  [0,  t],  In  the  literature  the  Z  process  is  called  an  SMP. 
However,  the  MRP  and  the  SMP  are  different  aspects  of  the  same  under¬ 
lying  stochastic  process;  therefore,  by  slight  abuse  of  language,  we 
shall  refer  to  the  underlying  process  itself  as  an  SMP  or  a  MRP,  using 
the  terms  interchangeably. 

A  MRP  is  regular  if  with  probability  one  (w.p.l)  each  state  is 
entered  only  a  finite  number  of  times  in  any  finite  time  span--i.e., 
if  P[N.(t)  <  «1  ■  1,  Vi  (  I  and  t  a  0.  A  MRP  is  strongly  regular 
if  w.p.l  the  total  number  of  state  transitions  is  finite  in  any 
finite  time  span--i.e.,  if  P[Z  N^(t)  <  ■]  «  1,  Vt  i  0.  Clearly 
strong  regularity  implies  regularity  and,  if  n  <  <*>,  the  terms  are 
equivalent.  In  applications,  it  will  ordinarily  be  obvious  that 
strong  regularity  holds.  If  n  <  ®,  it  suffices  that  H  *  (Hq,  ...»  H^) 
have  at  least  component  nondegenerate  at  zero  for  every  ergodic  sub¬ 
chain  of  the  imbedded  Markov  chain.  In  the  denumerable  state  case 
(n  -  •),  see  Pyke  [26]  and  Pyke  and  Schaufele  [27]  for  conditions 
that  imply  strong  regularity.  In  the  sequel,  we  assume  that  strong 
regularity  holds. 
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IV.  FIRST  PASSAGE  AND  COUNTING  DISTRIBUTIONS 


I.et 


Ptj(t)  -  P[Z(t)  -  j |Z(0)  -  l] 

CtJ(t)  -  PfN^t)  >  0 1 Z (0)  -  1] 

(first  passage  time  distribution) 


M^Ct)  -  E(N^ (t)  |Z(0)  -  i] 

(mean  entry  counting  function). 


Defining  the  convolution 


(A  *  B)(t) 


x)  dB(x), 


we  have  by  straightforward  renewal-theoretic  arguments: 


ij 


(1  - 


V»u 


♦Ei 

k 


ik 


*  P, 


(1  - 


Hi>fiij 


+PJJ*C 


i) 


CU  *  +  £  5lk  *  Cki 


Mu  ■  cij  +  °u  *  ")J  ■  Qu  +?Qtk  *V 


In  general,  these  relations  cannot  be  solved  analytically.  In  the 
finite  state  case,  numerical  solutions  can  be  obtained  by  numerical 
inversion  of  the  corresponding  Laplace  transforms  (see,  e.g. ,  [ 2  3, 
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[18],  [24],  and  [30]).  For  each  value  of  s,  only  one  matrix  inversion 
in  the  transform  domain  is  required--that  of  1  -  q(s),  where  s  >  0  and 


In  obvious  notation,  having  found  [I  *  q(s)]  * ,  either  analytically 

4 

as  a  function  of  s  or,  for  suitably  spaced  values  of  s,  numerically, 
one  successively  computes 

m(s)  -  [I  -  q(s)]  *q(s)  -  [I  -  q(s)]*1  *  I 

giJ(s)  -  mi j(s)/[l  +  “jj(8)1 
Pij(s)  *  Pjj(8)gij(a) ,  i  i  j 

1  -  h j (s) 

P)J(8>  *  1  -  gjj(s)  * 

and  then  inverts  the  transforms.  Although  this  procedure  is  not 
trivial,  it  often  compares  favorably  with  the  alternative  simulation 
approach  for  getting  the  transient  behavior  in  the  time  domain.  By 
usual  limit  theorems  for  l.aplace  transforms  (Widder  [31],  Feller  [12], 
see  also  Jewell  [20]),  the  behavior  in  the  time  domain  for  large 
(small)  t  corresponds  to  behavior  in  the  transform  domain  for  small 
(large)  s. 

Conditioning  on  the  event  that  no  state  in  a  subset  B  of  I+  is 
entered  in  (0,  t]  may  be  of  interest.  For  example, 
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B  P^CO  -  P[Z(t)  -  J|Z(0)  -  i,  Nk(t)  -  0,  Vk  €  B] 

B  Glj<t>  “  P[N j  (t)  >  0|Z(0)  -  i,  Nk(t)  -  0,  Vk  €  B") 
B  M^Ct)  -  ECNj(t)  |Z(0)  -  i,  Nk(t)  -  0,  Vk  €  B] 


£  Pikkj2>  +  2vik^j]  +  vi2>  * 

Krj  L  - 


where  is  the  mean  of  ,  v'  ■»  v,  and 


f. 


We  assume  that  v[2^i  <  •»  Vi  6  I+.  If  the  imbedded  Markov  chain  is 

★ 

finite  and  ergodic,  these  equations  have  a  unique  finite  solution 
and,  with  rr^  the  stationary  probability  that  the  last  state  entered 


*See  appendices  1  and  2  of  Fox  [13]  for  an  efficient  way  to  solve 
these  equations.  (An  expression  for  the  "bias  terms"  in  Markov  renewal 
prosjramming  involves  the  first  passage  time  moments,  which  are  of 
intrinsic  interest,  but  recently  Jewell  [22]  has  derived  a  remarkably 
simple  alternative  expression,  obviating  the  need  to  calculate  fu^ j ) 
and  {n[2)}  to  evaluate  the  bias  terms). 
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*  *  (2) 

is  J  if  all  were  degenerate  at  one,  multiplying  and 
by  rr^  and  summing  yields 


*jj  *  (1/nj)  E"kvk 
k 


“jf  "  <1/TTj)fcnkvk2)  +  2  E  nipikvik^j  ■ 

L  K  krj  J 


For  finite  state  SKP‘s,  the  probability  that  state  j  is  ultimately 

** 

reached  starting  from  i  is 


1, 

v->  -  <°- 


if  i.  i  6  \ 

if  i  €  j  €  E|,  k  i  j 


k-l 


[<1  *  A)  8]it  if  i  €  T,  J  € 

where  A  is  the  submatrix  of  P  corresponding  to  the  set  T  of  transient 

states.  E, ,  ....  E  are  the  recurrent  subchains  of  P,  and 
*  1  3  m  ’ 


*i  m  E  Pif.  i  €  T. 
1€  K 


The  mean  time  to  leave  T  starting  from  i  is 


5t  -  [(I  -  A)"V]t,  i  €  T  , 

where  is  the  vector  of  Vj's,  j  €  T.  A  double  generating  function 
for  the  distribution  of  Nj(t)  is 

★ 

In  other  words,  tt  is  the  stationary  measure  for  the  Imbedded 
chain,  but  not  (in  general)  for  the  SMP  itself. 

**The  case  i,  j  €  T,  of  less  interest,  is  not  considered. 
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Ya  -  i  •  (1  -  *)«  [a  1  +  (1  -  *)Df  *, 
where  ]  li  i  matrix  of  l's, 

D  ■  (<l '  ■  (t^n) 

and 

f*  *  •» 

9t  dtwiJ<*5  *> 

wiJ(x;  c)  *kviJ<k;  e) 

k-0 

vAJ  <k;  t)  -  P[Nj  (t)  -  k|z(0)  -  if. 

Thus,  in  principle,  Che  probabilities  and  moments  can  be  obtained  in 
the  usual  way.  The  Laplace  transform  m  of  the  first  moment  (M^(#)) 
was  already  given.  We  remark  that,  if  Z(0)  *  i  and  i  belongs  to  the 
same  ergodic  subchain  as  J ,  t  *Nj(t)  1/ujj  (1/*  *  0)  w.p.l,  a 

strong  law  that  follows  immediately  from  renewal  theory.  See  Pyke 
and  Schaufele  [28l  for  further  general  moment  computations,  weak  and 
strong  laws  of  large  numbers,  and  central  limit  theorems. 
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V.  STATE  CLASSIFICATION 

In  classifying  Che  states  of  a  SMP  transient,  null  recurrent,  or 
positive  recurrent,  we  must  distinguish  between  a  state's  classifl- 
cation  in  the  Imbedded  Markov  chain  and  in  the  SMP  Itself.  For  I 
finite  and  <  ®,  Vi  €  I+,  the  distinction  disappears  and  a  state  j 
is  either  transient  or  positive  recurrent  ■  1  and  <  •) . 

4 

In  large-scale  applications,  the  ergodic  subchain-transient  set 
breakdown  may  not  be  obvious  and  recourse  may  be  necessary  to  an 
algorithmic  classification  scheme  such  as  that  of  Fox  and  Landl  [16]. 

For  I  Infinite,  a  state  j  is  transient  (recurrent--!. e. ,  “  1) 

in  the  SMP  »  j  is  transient  (recurrent)  in  the  imbedded  Markov  chain. 
State  l  is  positive  recurrent  in  the  imbedded  Markov  chain  (contained 
in  ergodic  subchain  E^)  and,  for  some  constant  c,  £  c  <  ®,  Vi, 

)  £  E^,  «*  l  is  positive  recurrent  in  the  SMP.  A  SMP  is  positive 

recurrent  if  all  the  states  in  I+  are  positive  recurrent  in  the  SMP. 
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VI,  STATIONARY  PROBABILITIES 


It  is  important  to  distinguish  between  the  stationary  proba- 
billtles  fn^ ]  with  respect  to  the  imbedded  Markov  chain  and  the 
stationary  probabilities  {p^  with  respect  to  the  SMP.*  Thus  Pj, 
is  the  steady  state  probability  that  the  last  distinguished  state 
entered  is  i.  Hence  the  are  of  direct  interest  in  applications, 

while  the  are  computed  only  as  an  intermediate  step.  We  consider 

first  the  case  I+  finite  and  <  •,  Vi  €  I+. 


V“jr 

0, 

0, 


j  €  Ek,  2(0)  C  Ek 
J  €  Ek,  2(0)  -  i  €  T 
J  €  T 

j  €  2(0)  €  Er  k  i*  | 


where  G 


(•)  was  computed  already  and 


V»jj 


"jvi 


E 

l€E, 


nivi 


with  fn^]  here  being  the  stationary  probabilities  for  the  imbedded 
Markov  chain  given  that  2(0)  ?  Ek- 

In  the  remainder  of  this  section  we  assume  that  the  imbedded 
Markov  chain  is  irreducible  and  that  the  SMP  is  positive  recurrent, 
where  l"*"  may  be  finite  or  infinite.  We  also  assume  that  the  mean 


*In  general,  the  stationary  probabilities  must  be  interpreted 
as  Cesaro  limits.  If  the  process  is  aperiodic,  these  reduce  to 
orvirary  limits. 
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transition  times  are  uniformly  bounded  away  from  zero;  i.e.  ,  0  <  «  < 
<  a.  With  these  assumptions,  Fabens  [101  »bows  that 


in  agreement  with  result  given  above  for  the  I*  finite  case.  Define 
c(x)  as  the  time  of  the  last  transition  before  or  at  x  and  t(x)  as 
the  time  of  the  next  transition  after  x.  The  random  variables 


Y(x)  ■  t(x)  -  x  (excess  r.v.) 
5(x)  *  x  -  o(x)  (shortage  r.v.) 


are  of  Interest.  Adding  to  the  previous  assumptions  the  hypotheses 
that  the  mean  recurrence  times  are  finite  and  that  Z(*)  is 

aperiodic,  Fabens  shows  that 

lim  P[6(t)  S  xjz(t)  -  i)  -  lim  P{Y(t)  <  x|Z(t)  -  i} 


[1 


H.(u)]  du. 


This  generalizes  the  well-known  result  from  renewal  theory  for  the 
one  state  case,  obtained  there  as  a  corollary  to  the  key  renewal 
theorem;  see,  e.g.,  Barlow  and  Proschan  [1]. 

The  general  question  of  existence  and  uniqueness  of  stationary 
measures  is  dealt  with  in  Pyke  and  Schaufele  [29].  Cneong  [3]  gives 
conditions  under  which  convergence  to  the  steady  state  is  geometric. 
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VII.  EXAMPLE:  THE  M/C/1  QUEUE 

To  Illustrate  the  notion  of  stationary  probabilities  for  a  SMP, 

we  consider  the  M/G/l  queue.  Let 

n.  “  the  stationary  probability  that  1  customers  are  in 
the  system  just  after  a  random  service  completion 
epoch 

0  •  the  stationary  probability  that  i  customers  are  in 

the  system  just  after  the  service  completion  epoch 
preceding  a  random  point  in  time 

p^  *  the  stationary  probability  that  i  customers  are  in 
the  system  at  a  random  point  in  time. 

We  assume  that  the  traffic  intensity  \b  Is  less  than  one,  where  \  is 

the  arrival  rate  and  b  the  mean  service  time,  assumed  positive. 

Although  it  is  easily  shown  that 

Xbn^,  i  a  1 

(1  +  Xb)nQ  -  1  -  (Xb)2,  i  -  0, 

it  turns  out  that  pA  ■*  rr^ ,  Vi,  a  remarkable  result  originally  due  to 
Khintchine  [23]  and  derived  in  a  more  elementary  manner  by  Fox  and 
Miller  [17]  using  SMP  theory.  A  similar  result  holds  for  the  G/M/l 
queue,  but  in  bulk  queues  (Fabens  [11]),  for  example,  the  stationary 
measures  for  the  imbedded  Markov  chain  and  the  original  queueing 
process  are  different. 

Readers  familiar  with  queueing  theory  may  prefer  to  skip  to  the 
last  paragraph  of  this  section.  In  between,  the  standard  manipulations 
yielding  G  (z) ,  the  generating  function  of  the  fn. },  are  performed. 

Recalling  that  state  n  means  that  there  are  n  people  in  the 
system  and  a  service  has  just  been  completed,  we  obtain  the  well  known 


transition  matrix  for  the  imbedded  Markov  chain: 
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To  obtain  the  stationary  vector  n  for  the  chain,  we  multiply  the  i-th 
relation  determined  by  ttP  ■  n  by  Z*  and  sum,  define  the  generating 
function 

G„<z>  "  Z  v1* 

i 

and  obtain  from  the  special  form  of  P  for  this  chain  by  an  easy  calcu¬ 
lation  the  standard  result 


CTT(*> 


ttq(1  -  a)Gk(z) 
Gk(z)  •  Z 


Using  the  fact  that  lim  G  (z)  ■  1  (i.e.,  the  probabilities  sum  to  1) 

z-*l”  n 

and  applying  L'Hospital's  rule, 


TTq  -  1  -  Xb. 


Summarizing  our  results  so  far, 

nQ(l  -  z)P(X(l  -  z)) 

Gtt(z)  P(X(1  -  zl)  -  z 

Thus  the  mean  number  in  the  system  averaged  over  service  completion 
2 

epochs  is,  with  a  the  variance  of  the  service  times, 


lim  G  (z)  *  n,  lim 

2— r  n  z-r 


G^(z)  -  2G^(z)(G^(z)  -  1) 
2<G^(z)  -  l)2 


and  by  the  fact  that  ni  “Pi*  Vi,  *8  also  the  mean  nu®ber  in  the 
system  at  random  point  in  time  (in  the  steady  state).  Higher  moments 
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and  piobabilit its  can  be  obtained  from  the  generating  function  by 

appropriate  differentiations,  which,  however,  become  quite  tedious. 

Having  found  G^(z) ,  the  stationary  waiting  distribution  for  the 

first  come,  first  served  (FIFO)  discipline  can  easily  be  found.  If 

an  arrival  finds  the  system  empty,  the  conditional  wait  in  queue  is 

0.  Otherwise  it  is  governed  by  the  remaining  processing  time  of  the 

customer  in  service,  the  excess  random  variable,  plus  the  service 

times  for  the  customers  (if  any)  already  in  queue.  Noting  that 

C  (z)  “  G  (z) ,  the  interested  reader  can  readily  derive  the  Pol.aczek- 
p  ff 

Khintchine  formula  for  the  stationary  wait  in  queue.  (The  Laplace* 

Stleltjes  transform  of  the  stationary  queueing  delay  distribution  is 

found  to  be  ttq  +  [Gp(p(s»  -  nQ].)*  See  Feller  [12],  p.  392, 

for  an  alternate  elegant  derivation  that  bypasses  the  calculation  of 

G  (z).  A  third  derivation  follows  from  the  fact  that  the  number  of 
P 

customers  in  the  system  just  after  a  departure  is  the  number  of 
arrivals  during  his  total  wait  (queueing  time  plus  service  time); 
the  resulting  equation  is  solved  by  taking  generating  functions 
yielding  the  standard  form  of  the  Laplace-Stlelt Jes  transform  of  the 
stationary  queueing  delay  distribution  stro/(s  -  \(1  -  P(s)),  a  version 
of  the  Pollaczek-Khintchine  formula.  For  a  fourth  derivation,  where 
the  (superfluous)  assumption  of  a  continuous  failure  distribution  is 
tacitly  made,  see  Cox  and  Miller  [S],  pp.  241-242. 

•Jf 

Comparison  with  the  standard  form  of  the  transform  yields  an 
interesting  and  surprising  identity.  The  derivation  depends  on  the 
fact  that,  since  the  arrival  process  is  Poisson,  an  arrival  plays 
the  role  of  a  random  observer. 


t 


i 
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[I.  ASYMPTOTIC  FORM  OF  M. 


From  a  previous  section,  we  know  that  for  finite  state  SMP's 


j  (s)  “  «i j  <®>C 1 


®i j  <s> 


r, » 


Formally  expanding  e  in  a  Taylor  series  and  integrating  termwise 
yields  for  1,  j  in  the  same  ergodic  subchain 


»,.<«)  -  j-  -  ^  +  0(1), 

11  *“1)  2Ujj2  “)) 


whence  by  a  Tauberian  argument 


t  _ „  Hjj 

Mij(t>  '  ^  (Cesaro)  ^2  '  ^ 


a  result  that  can  be  obtained  by  analogy  with  renewal  theory  for 
delayed  recurrent  events,  where  the  time  to  the  first  "renewal"  has 
distribution  and  the  spacing  between  subsequent  renewals  has 
distribution  Gy  If  the  SMP  is  aperiodic,  the  Cesaro  limit  reduces 
to  an  ordinary  limit.  It  can  be  shown  that  the  formal  manipulation 

used  to  obtain  the  asymptotic  expansion  of  m^j(s)  is  justified 

(2)  + 
provided  that  <  ®,  Vi  €  I  • 
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IX.  FINITE  SMP'S  WITH  COSTS 

Often  In  applications,  costs  are  associated  with  the  transitions. 
Measuring  time  from  the  start  of  an  i  j  transition,  let  C^j(x|t)  be 
the  cost  incurred  up  to  time  x  given  that  the  transition  length  is  t. 
The  expected  discounted  cost  for  a  transition  starting  from  state  i 
is  then 

m 

V*>-£p 

where  a  cost  Incurred  at  time  x  is  discounted  by  the  factor  e  "" . 

An  elementary  renewal  type  argument  then  shows  that  v^Cor),  the  total 
expected  discounted  cost  over  an  infinite  horizon  starting  from  state 
i,  satisfies 

v(af)  -  V(or)  +  q(or)v(or), 

where  or  >  0  and  v(or)  and  •y(a)  are  the  vectors  with  component',  v^ar) 
and  y(a) ,  respectively.  Thus,  assuming  a  finite  number  of  states, 

v(or)  "  [I  -  q(or)]*1Y(»). 

Since  I  -  q(0)  is  singular  and  a  direct  asymptotic  expansion  is  not 
obvious,  it  is  convenient  to  make  use  of  the  relation  between  q  and 
m  given  earlier  to  study  the  behavior  of  v(cr)  as  9  -  0+.  Following 
Jewell  [21],  we  have 


v(or)  ■  [I  +  m(or)  ]Y(o) . 
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Making  use  of  the  expansion  of  m(or)  given  in  the  preceding  section, 
we  find  that,  if  i  is  a  recurrent  state,  v^(o)  lias  the  form 

Vj.Cer)  ■  ijo  +  +  o(l) 

and  a  straightforward  argument  in  Fox  [13]  then  shows  that  this  form 
is  valid  tor  any  state;  i.e., 

v (cr)  *  i/a  +  w  +  o(l)  , 

where  expressions  for  1,  the  loss  rate  vector,  and  w,  the  bias  term 
vector,  can  be  found  in  Jewell  [21,  22]  and  Fox  [13],  where  appropriate 
conditions  are  given  to  justify  the  expansion.  Substituting  this 

relation  into  v(a)  *  V(a)  +  q(er)v(or)  and  equating  the  coefficients 

- 1  * 
of  a  and  the  constant  terms,  respectively,  yields 

Pj l-l 

y  +  Pw  *  w  +  y 

yi  “EPijvij*j 

■  viii,  if  i  is  recurrent. 

These  expressions  can  be  solved  uniquely  for  1,  but  w  is  determined 
only  up  to  an  additive  constant  in  each  ergodic  subchain;  see,  e.g., 

*This  procedure  can  be  justified  by  a  simple  contradiction 
argument.  Note  that  q^(a)  **  Pij(l  -  oVjj)  °(or)  and  tha*-  the 
loss  rate  for  all  states  in  an  ergodic  subchain  is  the  sairi. 
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Denardo  and  Fox  [8"|.  An  Interesting  and  intuitive  result  that  follows 

easily  from  the  above  formulas  is  that  the  loss  rate  for  each  state 

00  OO 

in  an  ergodic  subchain  is  the  same  and  equal  to  n'  'V'/ti tv  'v't 
(k) 

where  n  is  the  stationary  vector  for  the  corresponding  submatrix 
and  V*  and  v*  are  the  restrictions  V  and  y,  respectively,  to  E^. 

The  loss  rates  for  the  transient  states  are  obtained  from  the  fact 
reflected  in  Pi  ■  t  that  the  loss  rate  for  a  state  is  given  by  the 
appropriate  convex  combination  and  that  I  -  A,  where  A  is  the  sub¬ 
matrix  corresponding  to  the  transient  states,  is  invertible. 

Denoting  the  undiscounted  loss  up  to  time  t  by  L(t),  we  obtain 
from  the  asymptotic  expansion  of  v(er)  that 

L(t)  "  lt  (Cesaro)  W‘ 
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X.  MARKOV  RENEWAL  PROCRAMMING 


The  situation  becomes  more  Interesting  when,  at  each  state  1, 
one  has  a  set  of  options  and  the  choice  at  i  simultaneously  deter¬ 
mines  pjj,  F^j,  and  for  all  J  €  I+.  The  goal  Is  either  to  choose 
a  policy  that  minimizes  either  the  expected  discounted  loss  or  the  loss 
rate.  In  the  latter  case,  an  appropriate  secondary  objective  is  to 
minimize  (/,  w)  lexicographically,  which  is  especially  important  when 
some  policies  can  have  transient  states.  With  either  criterion,  an 

optimal  policy  can  be  found  by  linear  programming  when  I+  and  X  _.+  A. 

t  £1  i 

are  finite.  For  details,  see,  e.g.,  Jewell  [21,  22],  Fox  [13"),  Denardo 
[6,  7*| ,  and  Denardo  and  Fox  [8"),  where  references  to  the  earlier 
(extensive)  literature  on  the  subject  are  given.  Some  papers  (e.g., 
Derman  [10])  treat  the  I+  infinite  case,  but  the  author  believes  that 
for  applications  the  general  theory  developed  so  far  for  that  case  is 
inadequate  and  that  particular  problems  are  best  attacked  on  an  ad  hoc 
basis.  For  the  case  where  I+  is  finite  but  the  finiteness  restriction 
on  Xa^  is  dropped,  see  Fox  [14]. 
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XI.  ESTIMATION  AND  STATISTICAL  INFERENCE 

Moore  and  Pyke  [25]  develop  estimators  for  the  fp^}  and  the 
fFjj]  and  their  large  sample  distributions.  For  statistical  inference 
in  birth  and  death  queueing  models,  see  Wolff  [32].  Both  of  the  fore¬ 
going  approaches  are  objectivist,  i.e.,  non-Bayes ian .  When  a  large 
number  of  observations  are  at  hand,  the  objectivist  approach  is  un¬ 
objectionable  and  difficulties  stemming  from  a  possible  lack  of  con¬ 
sensus  of  prior  belief  do  not  emerge.  On  the  other  hand,  when  the 

observations  are  few  or  nonexistent,  as  is  common,  a  Bayesian  approach 

★ 

incorporating  prior  beliefs  and  loss  functions  is  essential.  Such  an 
approach  may  be  formal  or  may  simply  consist  of  a  sensitivity  analysis 
with  the  outcomes  being  given  subjective  weights.  In  the  realm  of 
decision  making,  policies  should  adapt  to  modified  beliefs  as  more 
observations  are  taken.  This  area  remains  largely  unexplored  and  is 
ripe  for  investigation. 

★ 

This  is,  of  course,  a  statement  of  the  author's  opinion.  These 
matters  are  highly  controversial. 

** 

In  this  connection,  [15]  may  be  of  interest. 


r 
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